.. code:: ipython3 from seeq import spy import pandas as pd # Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script spy.options.compatibility = 192 .. code:: ipython3 # Log into Seeq Server if you're not using Seeq Data Lab: spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False, quiet=True) Asset Trees 1: Introduction =========================== Asset trees are a foundational tool that can be used to wrangle the full analytic capabilities of Seeq’s software. They sort physical locations, pieces of equipment, and data on that equipment into a hierarchical structure. Organizing your data into an asset tree allows you to: - Utilize asset swapping to rapidly create identical visualizations for different pieces of equipment - Write high-value calculations for your components and scale them across all components in your tree - Automatically generate scalable content and custom analyses - Use your tree as a starting point for roll-ups, calculations, displays, dashboards, and reports In this notebook we will show how to define an asset tree using the SPy library, modify it to your liking, and push the resulting tree to Seeq Server. Defining an Asset Tree ---------------------- There are multiple ways to define an asset tree in SPy. All use the same function: ``spy.assets.Tree()``. We summarize each way here before taking a further look in the subsections below. 1. To create an empty tree, give a name for your new tree as input: ``spy.assets.Tree('My Tree')`` 2. To create a tree with custom structure defined by a CSV (comma-separated values) file, give the filename as input: ``spy.assets.Tree('my_folder/my_tree_template.csv')`` 3. To use an existing asset tree from Seeq as a starting point for your tree, give the name of that asset tree as input: ``spy.assets.Tree('Facility 12_Asset Tree')`` 1. Creating an Empty Tree ~~~~~~~~~~~~~~~~~~~~~~~~~ In many cases, the easiest and most efficient way to build an asset tree is to start with the name of your root asset and insert your items later. .. code:: ipython3 my_tree = spy.assets.Tree('My Tree', workbook='My Workbook') This tree will contain a single asset named “My Tree” to begin with. It is scoped to a new workbook named “My Workbook” that will be created when the tree is pushed to Seeq. Underneath the root asset, we will be able to freely insert more assets, signals, conditions, and scalars as we need. In the code example above, ``spy.assets.Tree`` is a *function*. The two pieces of text ``'My Tree'`` and ``'My Workbook'`` inside the parentheses are *inputs* to that function. In return, the function *outputs* a tree object, which we assign to a *variable* called ``my_tree``. This variable allows us to refer to the newly defined tree in future code. 2. Creating a Tree using CSV Files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Defining asset trees using CSV template files is a great option for those seeking to create large asset trees using minimal Python code, or for those who already have pre-defined hierarchies for their items that can be exported to CSV format. The file ``spy_tree_example.csv`` in the ``SPy Documentation/Support Files`` directory contains the following data: +--------------------------+-------------+-----------------+---------+--------------------+ | Name | Level 1 | Level 2 | Level 3 | Friendly Name | +==========================+=============+=================+=========+====================+ | Area A_Temperature | My CSV Tree | Cooling Tower 1 | Area A | Temperature | +--------------------------+-------------+-----------------+---------+--------------------+ | Area A_Relative Humidity | | | | Relative Humidity | +--------------------------+-------------+-----------------+---------+--------------------+ | Area B_Temperature | | | Area B | Temperature | +--------------------------+-------------+-----------------+---------+--------------------+ | Area B_Relative Humidity | | | | Relative Humidity | +--------------------------+-------------+-----------------+---------+--------------------+ | Area D_Temperature | | Cooling Tower 2 | Area D | Temperature | +--------------------------+-------------+-----------------+---------+--------------------+ | Area D_Relative Humidity | | | | Relative Humidity | +--------------------------+-------------+-----------------+---------+--------------------+ | Area E_Temperature | | | Area E | Temperature | +--------------------------+-------------+-----------------+---------+--------------------+ | Area E_Relative Humidity | | | | Relative Humidity | +--------------------------+-------------+-----------------+---------+--------------------+ When this file is given as input to ``spy.assets.Tree``, SPy will look at each row in the file, find an item in Seeq corresponding to the row, and place it into a newly created asset tree at the specified location. The columns of the file give SPy information about where to find the item and how to put it in your tree: - The **Name** column tells SPy what item to pull from Seeq Server. - The **Level** columns tell SPy where in the tree to put the item. - If a level column is empty for a particular row, then the rows above are referred to. For example, the third row above has levels *My CSV Tree*, *Cooling Tower 1*, and *Area B*. - The **Friendly Name** column tells SPy what what to call the item after putting it in the tree. - An optional **ID** column can help SPy find items from Seeq Server that don’t have unique names. For example, the first row tells SPy to take the data from the existing signal “Area A_Temperature” in Seeq, create a new signal called “Temperature” containing the same data, and insert the new signal into the tree underneath the asset “Area A”. The asset “Area A” will subsequently be underneath “Cooling Tower 1”, which is underneath the root asset “My CSV Tree”. Below we create a tree from this CSV file, and then use the ``visualize()`` function to look at what the end result is. .. code:: ipython3 my_csv_tree = spy.assets.Tree('Support Files/spy_tree_example.csv', workbook='My Workbook') my_csv_tree.visualize() 3. Working with an Existing Asset Tree ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Say you have an existing asset tree in Seeq, perhaps from an external datasource like OSIsoft PI AF, and you wish to clean up the tree or add calculations to it for further analysis. You can define a tree in SPy using this existing tree as a starting point by referring to it by name in the ``spy.assets.Tree`` input. For instance, let’s pull in the asset tree that organizes Seeq’s example data. Its root asset is named ``'Example'``. .. code:: ipython3 example_data_tree = spy.assets.Tree('Example', workbook='My Workbook', description='My custom copy of Example Data') This creates a tree that is a full copy of the example tree, with an added description so you can tell your new tree apart from the old tree. Therefore, when we make modifications and push them to Seeq, the original tree will remain unaltered. The only time a copy will not be made is if the tree we choose to work with was also created by SPy. An existing asset tree or subtree can also be pulled into SPy using the ID of its root asset… .. code:: python example_data_tree = spy.assets.Tree('656B88EC-E71F-44B6-B2A1-D60202B3B0CD') …or by using ``spy.search`` results. .. code:: python search_results = spy.search({'Name': 'Example', 'Type': 'Asset', 'Datasource Name': 'Example Data'}) example_data_tree = spy.assets.Tree(search_results) Inserting Items into the Tree ----------------------------- The next step to building your asset tree is to add more items to it. Inserting the data that makes up your tree can be broken down into roughly three distinct substeps: 1. Insert assets to give the tree structure 2. Add signals, conditions, and scalars from Seeq server into your new asset hierarchy 3. Insert calculations Each of these steps is done using the ``insert()`` function. The ``insert()`` function can be called directly on your tree like this: ``my_tree.insert(...inputs...)``. Your tree will be updated accordingly. 1. Inserting Assets ~~~~~~~~~~~~~~~~~~~ Assets are logical groups of signals, conditions, scalars, or even other assets. In most use cases, they represent physical locations, pieces of equipment, or organizational collections of other assets. The assets of our tree will form its backbone, while other items will usually be at the bottom of our tree, farthest from the root asset. To insert anything into the tree, we need to specify (1) what we are inserting and (2) where we want to insert it. We will always specify where we are inserting using the ``parent`` input – for example, write ``parent='Area A'`` as input to insert your data directly underneath the asset in your tree with name “Area A”. To insert assets into the tree, give a name to the ``children`` input. .. code:: ipython3 my_tree.insert(children='Cooling Tower 1', parent='My Tree') my_tree.visualize() You can also give a list of names to create multiple assets at once. .. code:: ipython3 my_tree.insert(children=['Area A', 'Area B', 'Area C'], parent='Cooling Tower 1') my_tree.visualize() Now we’re ready to organize our data in the tree hierarchy we have created. 2. Inserting Signals, Conditions, and Scalars from Seeq Server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To insert items from the server, we will need to give our tree information on where to find those items. One practical way to find this information is to query items from Seeq by name using the ``spy.search`` function. .. code:: ipython3 search_results = spy.search({'Name': 'Area A_Temperature', 'Datasource Name': 'Example Data'}) We can pass these search results directly to the ``children`` input of the ``insert()`` function. Additionally, we can use the ``friendly_name`` input to rename the signal as we insert it. .. code:: ipython3 search_results = spy.search({'Name': 'Area A_Temperature', 'Datasource Name': 'Example Data'}) my_tree.insert(children=search_results, friendly_name='Temperature', parent='Area A') my_tree.visualize() The new signal “Temperature” we see underneath “Area A” is a copy of the existing signal “Area A_Temperature” from Seeq. The :doc:`example notebook for spy.search ` contains many more examples of how to query the server for the items you want to put in your tree. For instance, we can use `wildcards `__ to search for and insert many signals at once underneath Area B. .. code:: ipython3 search_results = spy.search({'Name': 'Area B_*', 'Type': 'Signal', 'Datasource Name': 'Example Data'}) my_tree.insert(children=search_results, parent='Cooling Tower 1 >> Area B') my_tree.visualize() Note that we included part of the path of “Area B” in the ``parent`` input – if there were multiple assets in the tree named “Area B”, the function would then know to insert under the Area B that lies underneath Cooling Tower 1. Using ``spy.search``, we can add hundreds or thousands of items to the tree in an organized manner. In addition to ``visualize()``, you can use the function ``missing_items()`` to verify that your tree is balanced. For example, it flags to us that the asset “Area F” in the example data is missing signals that other assets in Cooling Tower 2 have. .. code:: ipython3 example_data_tree.missing_items() Other attributes and functions helpful for understanding your tree include ``my_tree.size``, ``my_tree.height``, ``my_tree.name``, and ``my_tree.summarize()``. An alternative to using ``spy.search`` when inserting a single item is to copy the item’s ID in Workbench and pass that ID to the ``children`` input. .. code:: python my_tree.insert(children='219C06CC-67C5-4512-9242-2271596CEF56', parent='Area A') 3. Inserting Calculations ~~~~~~~~~~~~~~~~~~~~~~~~~ At this point our tree gives organization to pre-existing data. When we push it to Seeq, we can navigate around the tree and find the data grouped together by asset. While this functionality alone can enable powerful analytics in Seeq Server, we can further our analysis by creating *calculations* based upon the data items in our tree. A calculation requires a name, a formula, and a collection of formula parameters. The formula is written in `Seeq Formula Language `__, and the formula parameters assign variables in the formula to items in your tree. .. code:: ipython3 my_tree.insert(name='Too Hot', formula='$temp > 100', formula_parameters={'$temp': 'Temperature'}, parent='Area A') my_tree.visualize() It’s often handy to create the same calculation for many assets across your tree that contain similarly named signals. This can be done by using `wildcards `__ in the ``parent`` input, just like when using ``spy.search()``. Children will be inserted underneath every asset in the tree that matches the query given to ``parent``. Let’s try adding a “Dew Point” calculation underneath every area in a tree. Note that we’ll use ``my_csv_tree`` for this example instead of ``my_tree`` because its signals have more uniform names across the tree. .. code:: ipython3 my_csv_tree.insert(name='Dew Point', formula='$t - ((100 - $rh.setUnits(""))/5)', # From https://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html formula_parameters={'$t': 'Temperature', '$rh': 'Relative Humidity'}, parent='Area ?') my_csv_tree.visualize() Because there are signals “Temperature” and “Relative Humidity” under each parent asset, the calculations can successfully be inserted, with their formula parameters referring to the respective signals under their parent asset. The ``parent`` parameter will also accept ``spy.search`` results, an ID, or `regular expressions `__ for more advanced querying, though we won’t show examples here. Removing and Moving Items ------------------------- After working with your new tree, you may find that you need to make adjustments. You can use the ``remove()`` and ``move()`` functions to do this. These functions may also be useful for narrowing the focus of a large asset tree you have copied from Seeq. The ``remove()`` function takes a single input that specifies which items in the tree to remove. Let’s start by removing a single signal from ``my_tree`` .. code:: ipython3 my_tree.remove('Area B_Compressor Stage') my_tree.visualize() Note that if an asset is removed, then all of its children will be removed as well: .. code:: ipython3 my_tree.remove('Area B') my_tree.visualize() The input for ``remove()`` also supports paths, wildcards, and ``spy.search`` results. .. code:: python example_data_tree.remove('Area A >> Temperature') example_data_tree.remove('Cooling Tower 2 >> Area ? >> Optimizer') example_data_tree.remove(spy.search({'Path': 'Example >> Cooling Tower 1', 'Asset': 'Area C'})) The ``move()`` function pops data out of the tree and inserts it back in at another location. It takes two inputs: ``source`` and ``destination``. The ``source`` is what is being removed, and the ``destination`` is the new parent it should have when re-inserted into the tree. Let’s say that we accidentally insert “Area C_Temperature” into Area A of My Tree. .. code:: ipython3 my_tree.insert(children=spy.search({'Name': 'Area C_Temperature', 'Datasource Name': 'Example Data'}), parent='Area A') my_tree.visualize() Then we can easily amend our issue without starting over or rerunning commands by using the ``move()`` function: .. code:: ipython3 my_tree.move(source='Area A >> Area C_Temperature', destination='Area C') my_tree.visualize() Pushing a Tree -------------- The last step to working with an asset tree in SPy is to push your changes to Seeq! Up until now, all of our operations have only modified objects in Python — in order to view your tree in Workbench and share it with others, use the ``push()`` function to send it to the server. .. code:: ipython3 my_tree.push() my_csv_tree.push() Tada! The link displayed in the output will take you to the workbook that now contains your asset tree. Detailed Help ------------- All SPy functions have detailed documentation to help you use them. Just execute ``help(spy.)`` like you see below. .. code:: ipython3 help(spy.assets.Tree) Advanced Features ----------------- Inserting with Custom DataFrames ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For even more flexible insertions, you can provide any `Pandas DataFrame `__ as input to the ``children`` argument. The features of the Pandas library can be harnessed to customize the path, name, formula, and properties of every item you are inserting into your tree. .. code:: ipython3 new_signals = pd.DataFrame([{ 'Name': 'Relative Humidity', 'ID': spy.search({'Name': 'Area A_Relative Humidity', 'Datasource Name': 'Example Data'}).ID.squeeze(), 'Parent': 'Area A' }, { 'Name': 'Dew Point', 'Formula': '$t - ((100 - $RH.setUnits(""))/5)', 'Formula Parameters': {'$t': 'Temperature', '$rh': 'Relative Humidity'}, 'Parent': 'Area A' }]) my_tree.insert(new_signals) Inserting Metrics ~~~~~~~~~~~~~~~~~ Once your tree is defined with the data items you’re interested in, you can add metrics to create tabular calculations. Metrics usually have more inputs than other types of tree items and therefore must be defined using a DataFrame. The required properties are the Name, Type (‘Metric’), and a Measured Item. Like formula parameters for calculations, the Measured Item, Bounding Condition, and Thresholds can refer to other items in your tree. .. code:: ipython3 new_metrics = pd.DataFrame([{ 'Name': 'Dew Point Hourly Average', 'Type': 'Metric', 'Parent': 'Area A', 'Measured Item': 'Dew Point', 'Statistic': 'Average', 'Duration': '1h', 'Period': '1h', }, { 'Name': 'Overheating Severity', 'Type': 'Metric', 'Parent': 'Area A', 'Measured Item': 'Dew Point', 'Aggregation Function': 'percentile(95)', 'Bounding Condition': 'Too Hot', 'Bounding Condition Maximum Duration': '48h', 'Metric Neutral Color': '#FFFFFF', 'Thresholds': { 'HiHiHi#FF0000': 95, 'HiHi': 'Temperature', 'Hi': 85 } }]) my_tree.insert(new_metrics) Inserting Roll-up Calculations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Roll-up calculations* are a great way to evaluate summary statistics across multiple assets in your tree in order to monitor the health and performance of your assets. To insert a roll-up calculation, use the ``roll_up_statistic`` and ``roll_up_parameters`` inputs to the ``insert()`` function. .. code:: ipython3 my_csv_tree.insert(name='Average Temperature of All Areas', roll_up_statistic='Average', roll_up_parameters='Area ? >> Temperature', parent='Cooling Tower ?') my_csv_tree The resulting calculation is created by applying the function specified by ``roll_up_statistic`` to all parameters that match the string given by ``roll_up_parameters``. In this case, the roll-up ``Cooling Tower 1 >> Average Temperature of All Areas`` calculates the average of ``Area A >> Temperature`` and ``Area B >> Temperature``, and similarly for Cooling Tower 2, Area D, and Area E respectively. Inserting with References ~~~~~~~~~~~~~~~~~~~~~~~~~ When inserting many items into a tree at once, you may want each item you are inserting to have a different friendly name and a different parent. To achieve this quickly, you can pass references to DataFrame columns to the arguments ``friendly_name`` and ``parent`` of the ``insert()`` function. In the Seeq example data, there are a collection of signals with names of the format ``'Area A_Temperature'``. Column references allows you to insert all of these signals at once, where the signal ``'Area A_Temperature'`` has parent ``'Area A'`` with friendly name ``'Temperature'``, while the signal ``'Area D_Compressor Power'`` has parent ``'Area D'`` and friendly name ``'Compressor Power'``, and so on. First let’s remove all existing signals from ``my_tree``. .. code:: ipython3 my_tree.remove('Area ? >> *') Then let’s grab the signals from Seeq: .. code:: ipython3 search_results = spy.search({'Name': 'Area ?_*', 'Datasource Name': 'Example Data'}, order_by='Name') search_results.head(5) A column value reference has the following syntax: ``{{ ... } ... }``. The substring within the inner braces specifies which column of the above DataFrame to look for data in. If nothing is provided where the second ellipsis is, then the data from that column will be returned. For example the first row seen above would return ``kW`` when queried with ``{{Value Unit Of Measure}}``. However, if we only want some of the data in the column, then we can provide an expression with wildcards that matches the data and has parentheses surrounding the substring we wish to extract. For example, ``{{Name}Area ?_(*)}`` will return ``Compressor Power`` in the first row. Let’s see it in action: .. code:: ipython3 my_tree.insert(children=search_results, friendly_name='{{Name}Area ?_(*)}', parent='{{Name}(Area ?)_*}') my_tree.visualize() Below are more examples of column value extraction syntax, with the following input ``children`` DataFrame: ==================================== =========== ===== ================= ID Name Unit Facility Type ==================================== =========== ===== ================= B8FA62DA-5C42-4CA3-9C3F-E80E6E5AE990 Site A_Temp °F Research Facility 51D52461-A667-413A-AD9E-548FE16E601B Site B_Flow gal/s Factory ==================================== =========== ===== ================= +------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+---------------------------------+ | Syntax | Syntax Description | Output for first row | Output for second row | +==================================================================+==================================================================================================================================================================================================================================================================================================================================================+==============================+=================================+ | ``{{Name}}`` | Get the value of the "Name" column for that row. | Site A_Temp | Site B_Flow | +------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+---------------------------------+ | ``{{Name}Site ?_(*)}`` | From the "Name" column, get the value of the wildcard match after ``"Site ?_"``. | Temp | Flow | +------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+---------------------------------+ | ``{{Name}(*_Temp)}`` | Get the entire value of the "Name" column for that row if it matches ``"*_Temp"``. Note: this pattern does not match the second row. If such a pattern is used for ``parent``, then the row would not be inserted under any parent. If such a pattern is used for ``friendly_name``, its original name will be used. | Site A_Temp | N/A | +------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+---------------------------------+ | ``{{Facility Type}} {{Name}Site (?)_*}`` | Get the value of the "Facility Type" column for that row. Then a literal ``" "`` character. Then from the "Name" column, get the value of the single-char wildcard between ``"Site "`` and ``"_"``. | Research Facility A | Factory B | +------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+---------------------------------+ | ``Average of {{Name}(Site ?)_*} {{Name}Site ?_(*)} ({{Unit}})`` | A literal ``"Average of "``. Then from the "Name" column, get the value of the substring matching ``"Site ?"`` before the ``"_"``. Then a literal ``" "`` character. Then from the "Name" column, get the value of the wildcard match after ``"Site ?_"``. Then a literal ``" ("``. Get the value of the "Unit" column. Then a literal ``")"``. | Average of Site A Temp (°F) | Average of Site B Flow (gal/s) | +------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------+---------------------------------+ Pushing Large Trees ~~~~~~~~~~~~~~~~~~~ If you’ve created a large tree, it can take significant time to execute the ``push()`` command. You can speed it up by making use of a *metadata state file*. You just need to specify a unique filename like so: .. code:: ipython3 my_csv_tree.push(metadata_state_file='Output/asset_tree_tutorial_metadata_state_file.pickle.zip') If the cell above is executed multiple times, you’ll notice that the ``Push Result`` column includes ``Success: Unchanged``, which means SPy didn’t bother sending anything to the Seeq Server since nothing had changed. The ``spy.assets`` Submodule and Asset Tree Templates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Read on to the :doc:`next tutorial page on asset trees in SPy ` to learn how to use the ``spy.assets`` submodule to build asset trees out of templates. API Reference Links ------------------- - :py:class:`seeq.spy.assets.Tree`